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(54) Transmission of the region of interest of an image 



(57) Image data is transnrtitted between terminals, 
for example mobile phones. The data is accommodated 
in a frame of a preset frame during communication be- 
tween the temninals. An image, including a target image 



to be transmitted is captured. Adjustments are made so 
that the target image to be transmitted from the obtained 
image substantially fully fills the image frame. The ad- 
justed image data in the image frame is compressed and 
transmitted. 
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Description 

BACKGROUND OF THE INVENTION 
1 . Field of the Invention 



[00011 '^^^ present invention relates to a data trans- 
mission method, a data transmission apparatus and a 
data transmission system. 

[0002] Embodiments of the present invention relate 
to: a method for transmitting data including images dur- 
ing communication and streaming among a plurality of 
terminals; a data transmission apparatus such as a mo- 
bile phone, a PDA, a TV telephone tenninal, and a PC 
tenninal; and a data transmission system. 
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2. Description of the Related Art 

[0003] As a communication service for communica- 
tions and streaming among a plurality of temiinals, there 
is for example, as illustrated in Fig. 1 , the visual com- 
munication sen^ice for transmission of data and images 
by mobile terminals (MT) comprised of portable phones. 
[0004] The visual communication service system 1 
compresses voice and images and exchanges signals 
between a mobile terminal (hereinafter simply referred 
to as a terminal") 2-1 and tenninal 2-2 through a trans- 
mission channel 3 to enable conversation to be enjoyed 
while viewing the other partys face. 
[0005] Each of the tenninals 2-1 and 2-2. whose ap- 
pearances are illustrated in Fig. 1 . has a body case 21 , 
an operation panel 22 provided at the lower portion of 
the body case 21 and including a keypad for inputting a 
telephone number for a call function or for inputting cer- 
tain operational instructions, a display monitor 23 pro- 
vided above the operational panel 22, a camera 24 pro- 
vided above the display monitor 23 (at the upper portion 
of the body case 21 ), and an antenna 25. 
[0006] The transmission channel 3 includes mobile 
base stations (MBS) 31-1 and 31-2, mobile switching 
centers (MSC) 32-1 and 32-2, gateway mobile switching 
centers (GMSC) 33-1 and 33-2 having home location 
registers (HLR), and a circuit network 35. 
[0007] The mobile base stations 31 -1 and 31 -2 conv 
munlcate with the tenninals 2-1 and 2-2. The signals 
handled by the base stations 31 -1 and 31 -2 are sent to 
the higher level mobile switching centers 32-1 and 32-2 
and further to a backbone circjuit network 35 through the 
higher level gateway mobile switching centers 33-1 and 
33-2. 

[0008] When transmitting images to such a circuit net- 
work 35, however, the infonnation canying capacity of 
the circuit network 35 is limited by the bandwidth of the 
circuit network 35. 

[0009] Therefore, a user who is talking using for ex- 
ample the tenninal 2-1 can view his or her own image 
captured by the camera 24 at a high quality such as with 
the image shown by 23a in Fig. 1 because the image is 



monitored directly, but after compression at the tenninal 
2-1 and transmission to the terminal 2-2. the quality of 
the image sent from the tenninal 2-1 becomes extremely 
degraded at the tenninal 2-2 such as shown by the im- 
5 age shown by 23b. 

[0010] Similarly, at the tenninal 2-1 , the quality of the 
image sent from the tenninal 2-2 to the tenninal 2-1 be- 
comes extremely degraded. 

[0011] in this way, in an image transmission system 
10 of the prior art, the transmitted images become extreme- 
ly degraded. The difference from the image which the 
user himself or herself monitors becomes too great. This 
causes not only a strong uncomfortable feeling but also 
a feeling of stress as the expression of the other party 
15 cannot be seen clearly. 

[0012] Further, in each of the tenninals 2-1 or 2-2, the 
camera 24 is located above the display monitor 23 of 
the temilnal. The users look at their display monitors 23 
of their terminals. Therefore there is the problem that 
20 both end up looking down and therefore conversation 
maintaining eye contact becomes impossible. 

SUMMARY OF THE INVENTION 

25 {001 3] An embodiment of the present invention as de- 
scribed herein seeks to provide a data transmission 
method able to maintain the quality of a transmitted im- 
age substantially the same as that of one's own image 
and an apparatus using the method and a data trans- 
30 mission system. 

[001 4] An embodiment of the present invention as de- 
scribed herein seeks to provide a data transmission 
method able to realize communication in a state where 
eye contact is maintained and an apparatus using the 
35 method and a data transmission system. 

[0015] One aspect of the present invention provides 
a data transmission method for transmitting at least im- 
age data accommodated in an image frame of a preset 
size, comprising the steps of capturing an image includ- 
40 ing a target image to be transmitted; making adjust- 
ments so that the target image to be transmitted from 
the captured image substantially fully fills the image 
frame; and compressing and transmitting the adjusted 
image data in the image frame. 
45 [0016] Another aspect of the present Invention pro- 
vides a data transmission method for transmitting at 
least image data comprising the steps of capturing an 
image including a target image to be transmitted; mak- 
ing adjustments so that the target image to be transmit- 
so ted from the captured image has a preset size; and com- 
pressing and transmitting the adjusted image data in the 
image frame. 

[0017] The method of the present invention may fur- 
ther comprise, when making adjustments so that a tar- 
55 get image fully fills an image frame or when making ad- 
justments so that the target image to be transmitted from 
the captured Image has a preset size, using an image 
of a characteristic portion of the image to be transmitted 
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for matching and cutting out the image around the part 
with the best match for tracl<ing of the Image. 
[001 8] Alternatively, the method of the present Inven- 
tion may further comprise , when malcing adjustments 
so that a target image fully fills an Image frame or when 5 
making adjustments so that the target Image to be trans- 
mitted from the captured Image has a preset size, using 
an image of a characteristic portion of the image to be 
transmitted to calculate a distance in the characteristic 
portion and using this for enlargement, reduction, and 
tracking of the Image. 

[001 9] Alternatively, the method of the present Inven- 
tion may further comprise, when detecting a character- 
istic portion of an image, locking an image in a desired 
image state while monitoring a state of input of the im- 
age to be transmitted and using a center portion of the 
locked Image as the characteristic portion of the image. 
[0020] Altematlvely, the method of the present inven- 
tion may further comprise, when detecting a character- 
istic portion of an image, displaying a state of input of 
the image to be transmitted, specifying a certain location 
on the display screen, and using the portion around the 
specified point as the characteristic portion of the image. 
[0021] Alternatively, the method of the present inven- 
tion may further comprise, when locking a target image 
by using a characteristic portion of an image, fetching 
and tracking the image at a frame rate the same as or 
higher than the image to be transmitted and refreshing 
a reference image serving as a reference for tracking at 
a rate the same as or higher than the transmission frame 
rate. 

[0022] ... Altematlvely, the method of the present inven- 
tion may further comprise displaying a state of Input of 
a transmitted image and specifying a range to be trans- 
mitted on the display to determine the range of trans- 
mission of the image. 

[0023] A further aspect of the present invention pro- 
vides a data transmission method for displaying at least 
image data of a user on a display screen among a plu- 
rality of terminals and transmitting at least image data 
while displaying received Image data on the display 
screen, comprising capturing an image including a tar- 
get image of a user to be transmitted from substantially 
the center of the display screen. 
[0024] A yet further aspect of the present invention 
provides a data transmission method for displaying at 
least image data of user on a display screen among a 
plurality of terminals and transmitting at least image data 
while displaying received image data of another party 
on the display screen and displaying content for com- so 
mon discussion on the display screen, comprising the 
steps of capturing an image including a target image of 
a user to be transmitted from substantially a center of 
the display screen and displaying the image data atone 
of an upper portion or a lower portion of the display 55 
screen from the substantially center portion of the dis- 
play screen serving as the capturing portion and display- 
ing the content at the other portion. 



[0025] That is, the method of the present invention 
comprises displaying image data above the content on 
the display screen or displaying Image data below the 
content on the display screen. 

[0026] Still another aspect of the present invention 
provides a data transmission apparatus for transmitting 
at least image data accommodated in an image frame 
of a preset size, comprising an Imaging means for cap- 
turing an image Including a target Image to be transmit- 
ted; a first circuit for making adjustments so that the tar- 
get image to be transmitted from the captured image by 
the imaging means substantially fully fills the image 
frame; and a second circuit for compressing and trans- 
mitting the adjusted image data in the Image frame. 
[0027] Yet another aspect of the present invention 
provides a data transmission apparatus for transmitting 
at least Image data, comprising an Imaging means for 
capturing an Image including a target Image to be trans- 
mitted; a first circuit for making adjustments so that the 
target image to be transmitted from the captured image 
has a preset size; and a second circuit for compressing 
and transmitting the adjusted image data In the image 
frame. 

[0028] In an embodiment of the data transmission ap- 
paratus of the present invention, when making adjust- 
ments so that a target image fully fills an image frame 
or when making adjustments so that the target image to 
be transmitted has a preset size, the first circuit uses an 
image of a characteristic portion of the image to be 
transmitted for matching and cuts out the image around 
the part with the best match for tracking of the image. 
[0029] Alternatively, In an embodiment of the data 
transmission apparatus of the present invention, when 
making adjustments so that a target image fully fills an 
image frame or when making adjustments so that the 
target image to be transmitted has a preset size, the first 
circuit uses an Image of a characteristic portion of the 
image to be transmitted to calculate a distance In the 
characteristic portion and uses this for enlargement, re- 
duction, and tracking of the image. 
[0030] Alternatively, an embodiment of the data trans- 
mission apparatus of the present Invention comprises a 
displaying means for displaying image data and a lock- 
ing means able to lock an image displayed by the dis- 
playing means in a desired state, wherein when detect- 
ing a characteristic portion of an Image, the first circuit 
uses a center portion of the image locked by the locking 
image as the characteristb portion of the image. 
[0031] Alternatively, an embodimentof the data trans- 
mission apparatus of the present invention comprises a 
displaying means for displaying image data and includ- 
ing a pointer able to specify a certain location and a 
specifying means able to specify a certain location of 
the image displayed on the displaying means by thie 
pointer, wherein when detecting a characteristic portion 
of an image, the first circuit uses a portion around the 
point specified by the specifying means as the charac- 
teristic portion of the image. 
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[0032] Alternatively, in an embodiment of the data 
transmission apparatus of the present invention, when 
locking a target image by using a characteristic portion 
of the image, the first circuit fetches the image and per- 
fornis a tracking operation at a frame rate the same as 
or higher than the image to be transmitted and refreshes 
a reference Image serving as a reference for tracking at 
a rate the same as or higherthan the transmission frame 
rate. 

[0033] Alternatively, an embodiment of the data trans- 
mission apparatus of the present Invention comprises a 
displaying means for displaying image data and includ- 
ing a pointer able to specify a certain location and a 
specifying means able to specify a certain location of 
the image displayed by the displaying means by the 
pointer, wherein the first circuit decides on the range 
specified by the specifying means as the range for trans- 
mission of the image. 

[0034] A yet further aspect of the present Invention 
provides a data transmission system for displaying at 
least image data of a user on a display screen among 
a plurality of temiinals and transmitting at least image 
data while displaying received image data of another 
party on the display screen, wherein a temiinal compris- 
es an imaging means for capturing an image including 
a target image to be transmitted; a first circuit for making 
adjustments so that the target Image to be transmitted 
from the captured image by the imaging means substan- 
tially fully fills an image frame; and a second circuit for 
compressing and transmitting the adjusted Image data 
In the image frame. 

[0035] A further aspect of the present invention pro- 
vides a data transmission system for displaying at least 
image data of a user on a display screen among a plu- 
rality of terminals and transmitting at least image data 
while displaying received image data of another party 
on the display screen, wherein a terminal comprises an 
Imaging means for captu ring an image including a target 
image to be transmitted; a first circuit for making adjust- 
ments so that the target image to be transmitted from 
the captured Image becomes a preset size; and a sec- 
ond circuit for compressing and transmitting the adjust- 
ed image data in an image frame. 
[0036] A further aspect of the present invention pro- 
vides a data transmission system for displaying at least 
image data of a user on a display screen among a plu- 
rality of terminals and fortransmitting at least image data 
while displaying received image data on the display 
screen, wherein an imaging means for capturing an im- 
age including a target image to be transmitted is provid- 
ed substantially at the center of the display screen. 
[0037] A further aspect of the present invention pro- 
vides a data transmission system for displaying at least 
image data of a user on a display screen among a plu- 
rality of terminals and for transmitting at least image data 
while displaying received image data of another party 
on the display screen and displaying content for com- 
mon discussion on the display screen comprising an inn- 



aging means provided substantially at the center of the 
display screen and capturing an image including a target 
image to be transmitted and a displaying means for dis- 
playing the image data at one of an upper portion or a 
5 lower portion of the display screen from the substantially 
center portion of the display screen serving as the cap- 
turing portion and displaying the content at the other por- 
tion. 



[0038] According to embodiments of the present in- 
10 vention, in order for a target image such as an image of 
a user's face to fully fill an image frame, the face image 
is enlarged, reduced, or tracked according to its move- 
ment. Then, only the face image is cut out and that im- 
age is compressed and transmitted. 
15 [0039] Further, according to embodiments of the 
present invention, in order for the size of a user's trans- 
mitted Image to be kept constantly the same, the Image 
is enlarged, reduced, or tracked according to its move- 
ment. Due to this, an image of a person of the same size 
20 Is cut out and that image iscompressed and transmitted. 
[0040] Further, in an embodiment, by configuring the 
invention so that a user is able to lock (keep) an image 
in a desired state while viewing the monitor screen, the 
image can be tracked, enlarged, reduced, or cut out on 
25 the terminal according to the instructions of the user. 
[0041] Further, in an embodiment, in order to continue 
the locked state, at the time of starting locking, for ex- 
ample, a characteristic portion (a portion of the body 
such as the eyes, nose, mouth, ears, or outline of the 
30 face) is cut out from the image. Using this portion as a 
reference, the image is enlarged, reduced, or tracked. 
[0042] For example, the distance between the two 
eyes is detected, and from this distance, the rate of en- 
largement or reduction or the position of cut out Is de- 
35 tected 

[0043] In addition, the entire image input from the Im- 
aging means, that is, the camera, may be output for the 
image on the monitor of the user. The user specifies a 
region of the image to cut out that region. That image is 
40 then reduced, enlarged, or tracked so that this image 
subsequently remains the same size. 
[0044] Further, by using an operable pointer on a 
monitor screen, a user may specify a characteristic im- 
age at the time of starting locking and therefore that im- 
45 age is reduced, enlarged, or tracked. 

[0045] Further, In order to increase the tracking accu- 
racy, the image may be fetched and tracked at a frame 
rate the same as or higher than the image to be trans- 
mitted, and a reference image serving as a reference 
so for tracking may be refreshed at a rate the same as or 
higher than the transmission frame rate. 
[0046] Further, an Imaging means may be embedded 
in the display screen. Therefore, the lines of sight of the 
mutually transmitted images can be matched. 
55 [0047] In addition, according to an embodiment of the 
present invention, for example, a face image and con- 
tent for discussion are provided above and below each 
other and an imaging means is provided near the place 
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between them, so images of users talking while viewing 
the image under discussion can be transmitted to each 
other. Further^ virtual reality communication where eye 
contact can be maintained when viewing each others? 
faces on the monitor can be realized. s 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0048] A better understanding of the present invention 
will become apparent from the following illustrative de- 
scription of the preferred embodimentsf given with refer- 
ence to the attached drawings, in which: 

Fig. 1 is a view of terminals and a transmission 
channel in visual communication of the related art; 
Fig. 2 is a view of the system configuration of an 
embodiment of a visual communication service sys- 
tem as a data transmission system according to the 
present invention; 

Fig. 3 is a view of the appearance showing one ex- 
ample of the configuration of a mobile tenminat ac- 
cording to the present embodiment; 
Fig. 4 is a circuit diagram of a first embodiment of 
a signal processing system of a temninal as a data 
transmission apparatus according to the present in- 
vention; 

Fig.'S Is a circuit diagram of a pre-processing circuit 
In Fig. 4 according to the first embodiment. 
•Figs. 6A and 6B are schematic views for explaining 
the tracking in the pre-processing circuit; 
Figs. 7A to 70 are views for explaining a first meth- 
.od.fpr detennlnation of an initial reference image; 
Figs. 8A to 80 are views for explaining a second 
rmethpd for detemnlnatlon of an initial reference im- 
age; 

Fig. 9 is a circuit diagram of a second embodiment 
of a signal processing system of a temninal as a data 
transmission apparatus according to the present In- 
vention; 

Fig. 10 Is a circuit diagram of a pre-processing cir- 
cuit In Fig. 9 according to the second embodiment; 
Fig. 11 is a view for explaining an example of cal- 
culating a face size; 

Fig. 1 2 is a view for explaining an example of a face 
part detection search region; 
Fig. 13 is a view for explaining processing for re- 
ducing a face image in a face part detection search 
region and displaying the same on a display moni- 
tor; 

Fig. 14 is a view for explaining processing for re- 
ducing a face image in a face part detection search 
region and displaying the same on a display moni- 
tor; 

Fig. 15 is a view for explaining processing for en- 
larging a face image in a face part detection search 
region and displaying the same on a display moni- 
tor; 

Fig. 1 6 is a view for explaining processing for en- 



larging a face image In a face part detection search 
region and displaying the same on a display moni- 
tor; 

Fig. 1 7 Is a circuit diagram of a third embodiment of 
a signal processing system of a terminal as a data 
transmission apparatus according to the present in- 
vention; 

Fig. 18 is a circuit diagram of a pre-processing cir- 
cuit in Fig. 17 according to the third embodiment; 
Fig. 19 is a view for explaining processing for re- 
ducing a face Image in a specified an image trans- 
mission region and displaying it on a display monitor 
according to the third embodiment; 
Fig. 20 is a view for explaining processing for re- 
ducing a face Image in a specified an image trans- 
mission region and displaying it on a display monitor 
according to the third embodiment; 
Fig. 21 Is a view for explaining a ratio of pixels "e" 
of a face color over pixels "f" In an image on a mon- 
itor according to the third embodiment; 
Fig. 22 Is a view for explaining processing for re- 
ducing a face image in a face part detection search 
region and displaying it on a display monitor accord- 
ing to the third embodiment; 
Fig. 23 is a view for explaining processing for re- 
ducing a face Image In a face part detection search 
region and displaying it on a display nnon iter accord- 
ing to the third embodiment; and 
Fig. 24 is a conceptual view for explaining process- 
ing for fetching and tracking an Image at a frame 
rate the same as or higher than a transmitted image 
and refreshing a reference image for tracking at a 
rate the same as or higher than the transmission 
frame rate In order to increase a tracking accuracy 
of the image according to the fourth embodiment. 

DESORIPTION OF THE ILLUSTRATIVE 
ElVIBODIMErMTS 

[0049] Below, Illustrative embodiments of the present 
invention will be explained with reference to Fig . 1 to 24. 
[0050] Figure 2 Is a view of the configuration of an enrr 
bodiment of a visual communication service system as 
a data transmission system according to the present in- 
vention. 

[0051 1 The visual communication service system 1 00 
compresses voice and image data and exchanges sig- 
nals among a plurality of terminals, for example, be- 
tween a mobile temninal (hereinafter simply referred to 
as a "terminal") 50-1 and terminal 50-2 through a trans- 
mission channel 60. By this system, a user is able to 
enjoy conversation with another party while viewing his 
or her face. 

[0052] Figure 3 is a view of the appearance showing 
an example of the configuration of the temninal 50-1 or 
50-2. 

[0053] The terminal 50-1 or 50-2, whose appearance 
Is illustrated In Fig. 3, has a body case 51 , an operation 
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panel 52 provided at the lower portion of the body case 
51 and including a l<eypad for inputting a telephone 
number for a call function or for inputting specific oper- 
ational instructions, a display monitor 53 provided above 
the operational panel 52, a camera module 54 including 
a lens arranged at substantially a center region of the 
display monitor 53 and serving as an imaging means, 
an antenna 55. and a switch 56 acting as a locking 
means or specifying means and provided with functions 
as a GUI. 

[0054] In Fig. 3, a microphone and a speaker are omit- 
ted. The specific circuit configuration of a temninal acting 
as a data transmission apparatus will be described in 
detail later. 

[0055] In this way, the display monitor 53 has a lens 
portion of the camera module 54 arranged at the sub- 
stantially center region. As shown in Fig. 3. face images 
and a predetennined content image obtained from a lat- 
er explained application service provider (ASP) are dis- 
played above and below the position of the lens portion. 
[0056] In the example in Fig. 3. the face images are 
shown at the upper portion, and the contents image is 
shown at the lower portion , but it is also possible to con- 
versely display the content images at the upper portion 
and the face images at the lower portion. 
[0057] The transmission channel 60 includes mobile 
base stations (MBS) 61-1 and 61-2, mobile switching 
centers (MSG) 62-1 and 62-2. gateway mobile switching 
centers (GMSC) 63-1 and 63-2 having home location 
registers (HLR), and a circuit network 64. 
[0058] The mobile base stations 61 -1 and 61 -2 com- 
municates with the temninals 50-1 and 50-2, respective- 
ly. The signals handled by the base stations 61-1 and 
61 -2 are transmitted to the higher level mobile switching 
center 62-1 and 62-2 and further to a backbone circuit 
network 64 through the higher level gateway mobile 
switching centers 63-1 and 63-2. 
[0059] Further, a gateway (GW) 65 is connected to the 
circuit network 64, while a application service provider 
66 able to distribute content is connected with the gate- 
way 65. 

[0060] Therefore, it Is possible to start up the applica- 
tion service provider (ASP) 66 offering various content 
CONT and output the same images of the content 
CONT to the temninals 50-1 and 50-2 through the gate- 
way 65. 

[0061] Below, circuit configurations of illustrative sig- 
nal processing systems In terminals 50 (temninals 50-1 
and 50-2) acting as data transmission apparatuses ac- 
cording to the present invention, as applied to the above 
visual communication service system 100, will be ex- 
plained as first, second, and third embodiments with ref- 
erence to the accompanying drawings. 

First Embodiment 

[0062] Figure 4 is a circuit diagram of a first embodi- 
ment of a signal processing system of a temiinal func- 



tioning as a data transmission apparatus according to 
the present invention. 

[0063] The signal processing system 500, as shown 
in Fig. 4, has a microphone 501 . a camera circuit 502, 
5 a pre-processing circuit 503 as the first circuit, a video 
compressor and coder 504, a voice compressor and 
coder 505, a multiplexer (MUX) 506. a transmission cir- 
cuit 507, a reception circuit 508, a demultiplexer (De- 
MUX) 509: a video decoding circuit 51 0, a voice decod- 
10 ing circuit 511, an image combining apparatus 512, a 
display circuit 513, a speaker 514, a GUI interface (l/F) 
circuit 515, and a CPU 516. 

[0064] The video compressor and coder 504, voice 
compressor and coder 505. multiplexer (MUX) 506, and 
15 transmission circuit 507 fomn the second circuit. 

[0065] The microphone 501 converts the voice gen- 
erated by a user Into an electrical signal and outputs It 
to the voice compressor and coder 505. 
[0066] The camera circuit 502 perfonms certain 
20 processing on an image of for example the upper part 
of a user's body captured by the camera module 54 
through its lens and converted from an optical signal into 
an electrical signal and outputs the signal as a digital 
image signal 8502 to the pre-processing circuit 503. 
25 [0067] The pre-processing circuit 503 receives the 
digital image signal 8502, extracts an image to be trans- 
mitted (hereinafter referred to as a "target image") from 
the image of the upper part of the user's body, generates 
an image signal adjusted so that the image to be trans- 
30 mitted fully fills the image frame, and outputs it to the 
video compressor and coder 504 and image combining 
circuit 512. 

[0068] As the processing for adjusting the target im- 
age to fully fill the Image frame, the pre-processing cir- 
35 cult 503 for example uses the image of a characteristic 
portion of the image to be transmitted for matching and 
cuts out the images around the part of the best match 
so as thereby to enable tracking of the image. 
[0069] Here."f ully fill the image frame" means to thor- 
40 oughly fill It with the target image until the background 
is not recognizable. Especially, in image transmission 
apparatuses for communication, the target images are 
mostly face images. 

[0070] Below, the processing of the pre-processing 
45 circuit 503 will be described in further detail. 

[0071] Figure 5 Is a circuit diagram of an example of 
the specific configuration of a principal portion of the 
pre-processing circuit according to the first embodi- 
ment. 

50 [0072] In addition, Figs. 6A and 6B are schen»atic 
views for explaining the tracking processing by the pre- 
processing circuit. 

[0073] The pre-processing circuit 503, as shown in 
Rg. 5, comprises a search region extractor 5031 , a bi- 
55 nary-coded unit 5032, a pattern matching unit 5033. a 
reference memory 5034, and a rectangular region ex- 
tractor 5035. 

[0074] The search region extractor 5031 extracts a 
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search region SRG as shown in Fig. 6B based on the 
coordinate information of the reference image RIM of 
the previous Image PIM shown in Fig. 6A and stored In 
the reference memory 5034. 

[0075] The binary-coded unit converter 5032 converts 5 
the luminance signal of the search region data extracted 
by the search region extractor 5031 into binary data and 
outputs It to the pattern matching unit 5033. 
[0076] The pattern matching unit 5033 performs pat- 
tern matching between the luminance information of the 
binary data of the luminance signal of the search region 
given by the binary-coded unit 5032 and of the binary 
data of the reference Image RIM of the previous image 
PIM shown in Fig. 6A and stored in the reference mem- 
ory 5034 and outputs the results to the rectangular re- 
gion extractor 5035. 

[0077] The pattern matching processing employs the 
image data MMT at a position giving the highest con- 
sistency in the search region SRG as shown in Fig. 6B 
and uses it as the tracl<ing results of the current image 
CIM. 

[0078] In addition, the pattern matching unit 5033 
stores the traci<ing results at the present stage, that is, 
the image data MMT at a position giving the highest con- 
sistency, into the reference memory 5034 as a reference 
image for the next Image. 

[0079] The rectangular region extractor 5033 cuts out 
a certain rectangular region surrounding the tracking re- 
sults MMT obtained by the pattern matching unit 5033, 
and outputs the same. 

[0080] In the pre-processing circuit 503 having such 
a configuration, tracldng of the input image is perfomned 
in order to maintain a desired image state (hereinafter 
refen^ed^o.as an "Initial reference Image") that a user 
wishes to transmit. 

[0081] Then, the traclcing operation Is perfomried at a 
frame rate the same as or higher than the transmission 
frame rate, and a reference image serving as a refer- 
ence for tracking is refreshed successively. 
[0082] The tracking in the pre-processing circuit 503 
is started when infomnatlon of an Initial reference image 
(described later with reference to Figs. 7A to 7C and 
Figs. 8A to 8C) is transmitted from the CPU 51 6. 
[0083] First, in the input image data (the current im- 
age) CIM newly Input from the camera 502 as shown In 
Fig. 6B, a search region SRG is extracted by the search 
region extractor 5031 based on the coordinate informa- 
tion of the reference image RIM of the previous image 
PIM stored in the reference memory 5034. 
[0084] Next, the luminance signal of the extracted da- 
ta Is converted Into binary data In the binary-coded unit 
5033, then pattern matching of luminance infonnation Is 
performed by the pattern matching unit 5033 between 
the luminance signal of the extracted data and the bina- 
ry-coded data of the reference image RIM stored in the 
reference memory 5034. 

[0085] The tracking results at this time, namely, the 
image data MMT at a position giving the highest con- 
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sistency, Is stored in the reference memory 5034 as a 
reference image of the next Image. 
[0086] Then, the rectangular region extractor 5035 
cuts out and outputs a rectangular region surrounding 
the tracking result MMT obtained by the pattem match- 
ing unit 5033. 

[0087] Next, first and second methods for determina- 
tion of the initial reference image will be explained with 
reference to Figs. 7A to 7C and Figs. BA to 8C. 
[0088] The first method of determination detemilnes 
the image by monitoring the Image to be transmitted the 
Image size frame. 

[0089] in more detail, as shown in Fig. 7A. a transmis- 
sion image size frame FR Is set beforehand in the initial 
image IIM. The image in the frame FR is output as a 
monitor Image MNTa and, as shown in Fig. 78, is dis- 
played on the display monitor 53 by the displaying circuit 
513 In Fig. 4. 

[0090] The user views the output on the monitor and 
when a desired image state Is found, for example press- 
es the switch 56 to Issues a command for locking the 
image. Therefore, as shown In Fig. 7C, the initial refer- 
ence Image IRI is decided. 

[0091] On the other hand, the second method of de- 
temiination decides on the image by monitoring a re- 
duced image. 

[0092] In more detail, as shown in Fig. 8A, the initial 
Image IIM is reduced and output as a monitor image 
MNTb as shown in Fig. 8B. The reference center desired 
by the user and the display region (the region occupied 
by the Image) are specified using a pointer POI to decide 
the initial reference Image IRI. 

[0093] Note that, the Instruction for moving the pointer 
POI on the display monitor 53 Is executed by using for 
example a ball switch, that is, the switch 56. Namely, by 
rotating the switch 56 In a desired direction, the pointer 
POI is moved up, down, left, right, or in an inclined di- 
rection. 

[0094] Information on the initial reference image de- 
cided by the first or the second method in this way is 
sent from the GUI interface circuit 515 to the reference 
memory 5034 of the pre-processing circuit 503 through 
the CPU 516. 

[0095] Further, In Fig. 4, the video compressor and 
coder 504 compresses and codes the transmitted Image 
data extracted by the pre-processing circuit 503 and out- 
puts it to the multiplexer 506. 

[0096] The voice compressor and coder 505 com- 
presses and codes the voice data from the microphone 
501 and outputs it to the multiplexer 506. 
[0097] The multiplexer 506 multiplexes the image da- 
ta to be transmitted, which was compressed and coded 
by the video compressor and coder 504, and the voice 
stream compressed and coded by the voice compressor 
and coder 505 and outputs the data to the transmission 
circuit 507, 

[0098] The transmission circuit 507 converts and 
modulates the image and voice data multiplexed by the 
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multiplexer 506 into a specific transmission protocol and 
transmits it to the mobile base station (MBS) 61-1 or 
61-2. 

[0099] The reception circuit 508 receives and demod- 
ulates the transmitted signal sent from the other com- 
munication party through the transmission channel 60. 
decompresses the transmission protocol, and outputs 
the result to the demultiplexer 509. 
[01001 The demultiplexer 609 separates the image 
data and the voice data from the reception circuit 508 
and supplies the obtained image data stream and the 
voice data stream to the video decoding circuit 51 0 and 
the voice decoding circuit 511 , respectively. 
[01011 The video decoding circuit 510 decodes the 
Image data stream separated by the demultiplexer 509 
and outputs it to the image combining circuit 512. 
[01021 The voice decoding circuit 511 decodes the 
voice data stream separated by the demultiplexer 509 
and outputs it to the speaker 514 for reproduction. 
[01031 The image combining circuit 51 2 combines the 
face image data of a user of a terminal from the pre- 
processing circuit 503 and the face image data of for 
example the other party from the video decoding circuit 
510 or a specified content image from the application 
service provider (ASP) 66 and outputs the result to the 
display circuit 513. 

[01 041 The display circuit 51 3 receives the combined 
Image from the image combining circuit 512, displays 
on the display monitor 53 of the temiinal 50-1 (or 50-2). 
as shown in Fig. 3 and Fig. 4, a user's own face image 
and a transmitted face image of his other party for ex- 
ample side by side above the lens of the camera module 
54, and displays the specified content image below the 
lens. 

[01051 The GUI interface circuit 51 5 includes a switch 
56. The CPU 516 controls the entire temninal 50-1 (or 
50-2) according to the instructions from the user to shift 
to the image lock state, switch the display on the moni- 
tor, and so on. 

[01061 Next, the operation of the temninal 50 of Fig. 4 
will be described. 

[01071 For example, an Image of the upper part of a 
user's body is captured by the camera module 54 
through its lens. This is converted from an optical signal 
Into an electrical signal by the camera circuit 502 and 
output to the pre-processing circuit 503 as the digital im- 
age signal S502. 

[01 081 The image data of the upper part of the user's 
body is supplied to the display circuit 513 through the 
Image combining circuit 512. The image of the upper 
part of the user's body, that is, the initial image IIM, is 
then displayed on the display monitor 53. 
[0109] While viewing the image of the upper part of 
his or her own body, that is, the initial image IIM, the user 
decides on an initial reference image IRl by for example 
the first method, namely, monitoring the size frame of 
the image to be transmitted image. 
[01101 That is, the transmission image size frame is 



set beforehand in the initial image IIM. and the image in 
the frame FR is output as a monitor image MNTa and 
displayed on the display monitor 53 by the display circuit 
513. 

5 [01111 Then, the user views the output on the monitor 
and when finding a desired image state, for example 
presses the switch 56 to issues a command for locking 
the image to the CPU 51 6 to thereby decide on the initial 
reference image IRl. 
10 [01121 Then, when the initial reference Image infor- 
mation is transmitted from the CPU 51 6, tracking is start- 
ed in the pre-processing circuit 503. 
[01131 In this state, in the input image data (the cur- 
rent image) C t M newly input from the camera circuit 502 , 
15 a search region SRG is extracted by the search region 
extractor 5031 based on the coordinate infonmation of 
the reference image RIM of the previous Image PIM 
stored in the reference memory 5034. 
[01 1 41 Next, the luminance signal of the extracted da- 
20 ta Is converted into binary data by the binary-coded unit 
5033, then pattern matching of the luminance infomia- 
tion is perfonned by the pattern matching unit 5033 be- 
tween the luminance signal of the extracted data and 
the binary data of the reference image RIM stored in the 
25 reference memory 5034. This processing result is 
stored in the reference memory 5034 as a reference im- 
age of the next image. 

[01151 Then, the rectangular region surrounding the 
tracking result MMT obtained with the pattern matching 
30 unit 5033 is cut off in the rectangular region extractor 
5035 and it is output to the video compressor and coder 
504 and the image combining circuit 512. 
[01 161 *n such a state, the voice input from the micro- 
phone 501 is converted into digital and then is com- 
35 pressed by the voice compressor and coder 506. 

[01171 Then, the image data obtained from the pre- 
processing circuit as described above is compressed 
and coded by the image compressor and coder 504, 
multiplexed by the multiplexer 506 together with the 
40 compressed voice stream, and then output to the trans- 
mission circuit 507. 

[01 1 81 After that, the transmission circuit 507 receiver 
the multiplexed data, the transmission protocol, trans- 
mits the result for example to the mobile base stat.on 
45 61 -1 , and transmits it by the transmission channel 60 to 
for example the temninal 50-2 of the other party. 
[01191 In addition, in the reverse order, the transmit- 
ted signal from the other party is received by the recep- 
tion circuit 508. demodulated, has its transmission pro- 
50 tocol decompressed, then is separated into the image 
data stream and the voice data stream by the demulti- 
plexer 509. 

[01201 The separated voice data stream is decoded 
by the voice decoding circuit 51 1 and sent to the speaker 
55 51 4 for reproduction. 

[01 21 1 The image data stream Is decoded by the vid- 
eo decoding circuit 51 0. then is combined with the user's 
own image from the pre-processing circuit 503 by the 
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image combining circuit 51 2. The combined image data 
Is output to the display circuit 513 and displayed on the 
display monitor 53. 

[01 22] At this time, the display monitor 53 displays the 
user's own face image and a transmitted face Image of 
the other party are displayed for example side by side 
above the lens of the camera module 54 on the display 
monitor53 and displays a specified content Image isdis- 
played below the lens. 

[0123] As described above, according to the first em- 
bodiment, during communication between terminals 
50-1 and 50-2, the target image Is compressed and 
transmitted after being adjusted to fully fill the frame of 
the transmitted image, so the amount of infomriation can 
be greatly reduced while transmitting the expression of 
the other party, which is needed most for the communi- 
cation. 

[0124] In addition, because the face image is adjusted 
to fully fill the frame, transmission of the high resolution 
background image can be naturally eliminated, and the 
amount of information after compression can be largely 
reduced further. Consequently, it is possible to transmit 
an image of a high quality. 

[01 25] Further, the frame correlation of images can be 
increased greatly. By using a compression technique of 
interframe compression like MPEG for the image com- 
pression method, the amount of information can be dra- 
matically reduced and therefore it becomes possible to 
transmit an image of a high quality. 
[0126] Further, the image state that a user desires to 
transmit (image locking state) can be kept automatically. 
This helps detecting the characteristic portion of the im- 
age and enables tracking of the target image such as 
the image of the person. As a result, the time correlation 
of images is increased, and the amount of information 
after compression can be decreased, and so it becomes 
possible to transmit an image of a high quality. 
[0127] Further, since face Images and content for 
common discussion are arranged at the upper and lower 
portions of the display rrionitor 53 and since a lens part 
of the camera module 54 is provided between them, a 
state can be realized where the speakers are in better 
eye contact. 

[0128] Further, the state of the users talking while 
viewing an image for discussion can be transmitted. Fur- 
ther, there is an advantage that virtual reality communi- 
cation where the users maintain eye contact by looking 
at each others' faces on the monitor can be realized. 
[0129] Furthermore, in the transmission channel 60, 
an application service provider (ASP) 66 able to distrib- 
ute content may be started up and the same images of 
content CONT provided by the application service pro- 
vider 66 may be output to the terminals 50-1 and 50-2 
through the gateway 65. 

Second Embodiment 

[0130] Figure 9 is a circuit diagram of a second em- 
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bodiment of the signal processing system of a terminal 
functioning as a data transmission apparatus according 
to the present invention. 

[01 31 ] The point of difference of the signal processing 

5 system of a tenninal according to the second embodi- 
ment and the aforesaid signal processing system of ac- 
cording to the first embodiment lies in the processing of 
the pre-processing circuit 503A and the CPU 51 6A. 
[0132] Specifically, during communication, a part of 

10 the face is detected in the input Image, the size of the 
face is calculated from that information, the image is en- 
larged or reduced so that the face image fully fills the 
monitor, then the face image is cut out. 
[01 33] In this case, the Image of the characteristic por- 

15 tion of the image to be transmitted is used to calculate 
the distance of the characteristic portion and this used 
for enlarging, reducing, or tracking the image. 
[01 34] Particularly, In case of a face image, for exam- 
ple, the portion of the eyes is taken out for the charac- 

20 teristic portion, the rate of enlargement or reduction of 
the image is detemnined from the distance between the 
two eyes so that the distance becomes the same as that 
in a reference image, then the image is tracked. In ad- 
dition, as the characteristic portion, use may be made 

25 of at least part of the eyes, nose, mouth, ears, or outline 
of the face and the image enlarged, reduced, or tracked 
from their sizes and distances. 

[0135] The rest of the configuration is the same as that 
shown in Fig. 4. Therefore, in Fig. 9, components the 
30 same as In Fig. 4 are shown with the same reference 
numerals. 

[01 36] Namely, the signal processing system 500 A of 

a temninal, as shown in Fig. 9, has a microphone 501 , 
a camera circuit 502, a pre-processing circuit 503A, a 

35 video compressor and coder 504, a voice compressor 
and coder 505, a multiplexer (MUX) 506, a transmission 
circuit 507, a reception circuit 508, a demultiplexer (De- 
MUX) 509, a video decoding circuit 510, a voice decod- 
ing circuit 511, an image combining apparatus 512, a 

40 display circuit 513, a speaker 514, a GUI interface (l/F) 
circuit 515, and a CPU 51 6A. 

[0137] Because the functions of the pre-processing 
circuit 503A and the CPU 51 6A are different from those 
in the first embodiment, below, only the functions of the 
45 pre-processing circuit 503A and the CPU 51 6A will be 
described in detail. Descriptions of other components 
will be omitted. 

[01 38] Figure 1 0 Is a circuit diagram of an example of 
the specific configuration of the principal portion of the 
50 pre-processing circuit according to the second ennbod- 
iment. 

[01 39] For easy understanding, the CPU 51 6A Is also 
illustrated in Fig. 10. 

[0140] The pre-processing circuit 503A, as shown in 
55 Fig. 1 0, has a face part position detector 5036, an image 
enlarging and reducing unit 6037, an image extractor 
5038, and a reference face image memory 5039. 
[0141] The face part position detector 5036 detects 
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the position of parts of the face such as the two eyes, 
nose, mouth, eyebrows, ears, and the outline of the face 
within the infomiation on the initial range of the face po- 
sition by the CPU 51 6A, that is, a face part search re- 
gion, by using infonnation on luminance and colors and 
supplies the detected face part position information 
S5036 to the CPU 51 6A. 

[0142] The image enlarging and reducing unit 5037 
enlarges or reduces the input image based on the rate 
of enlargement or reduction calculated by the CPU 51 6A 
and supplies the same to the image extractor 5038. 
[01 43] The image extractor 5038 cuts out the face im- 
age in the region displayed on the monitor screen from 
the enlarged or reduced image based on the face posi- 
tion information (coordinates) already sent from the 
CPU 516A. 

[01 44] The cut out face Image is sent to the reference 
face Image memory 5039. At this time, a reference im- 
age is made from the rate of enlargement or reduction 
of the image. 

[01 45] Next, a description will be given of the opera- 
tion of the pre-processing circuit 503A, including the 
processing of the CPU 51 6A with reference to Fig. 1 1 to 
Fig. 16. 

[0146] Figure 11 isaviewof an example for explaining 
the calculation of a face size, Fig. 12 is a view of an 
example for explaining the search region for face part 
detection, Fig. 13 and Fig. 14 are views for explaining 
processing for reducing and displaying a face image in 
a search region for face part detection on a display mon- 
itor, and Fig. 15 and Fig. 16 are views for explaining 
processing for enlarging and displaying a face image in 
a search region for face part detection on adisplay mon- 
itor. 

[0147] First, forthe first Input image (the initial image). 
Information on the initial region of the face position is 
supplied to the CPU 51 6A by instructions by a user 
through the GUI Interface circuit 515 (switch 56). 
[0148] In the CPU 51 6A, the transmitted initial region 
is supplied to the face part position detector 5036 as a 
face part detection search region signal S516a. 
[01 49] In the face part position detector 5036. the po- 
sition of parts of the face such as the two eyes, nose, 
mouth, eyebrows, ears, and outline of the face is detect- 
ed In the supplied face part search region by using in- 
fonnation on luminance and colors. Then, the position 
infonnation on the face parts detected in the face part 
detector 5036 Is supplied to the CPU 51 6A as a signal 
S5036. 

[0150] In the CPU 51 6A. the size of the face in the 
image is calculated based on the position information of 
the face parts. 

[0151] For example, as shown in Fig. 11 , if denoting 
the distance between the detected two eyes as "a" and 
the perpendicular distance between the eyes and the 
mouth as "b", the size of the face is set to 3a In the hor- 
izontal direction and 4b in the vertical direction. 
[01 52] Note that in th is case, it is also possible to de- 



tect the color of the face to extract the contour and cal- 
culate the size of the face from that size. 
[01 53] Further, in the C PU 51 6A, the face part detec- 
tion search region of the next image is calculated based 
5 on the position information of the face parts. 

[0154] For example, as shown in Fig. 12, assume the 
region of ±a x N horizontally and +b x M and -b x L ver- 
tically around the rectangle a x b including the two eyes 
and the mouth is the face part detection search region 
10 of the next image. 

[0155] Note that the search region cannot be larger 
than the input image. 

[01 56] From the second input image and on, the face 
part detection search region obtained in the previous im- 
15 age is supplied to the face part position detector 5036 
by the CPU 51 6A. 

[01 57] In the face part position detector 5036 , the po- 
sition of face parts Is detected in the same way as above 
in the supplied search region. 
20 [0158] At this time, information of the face portion of 
the previous image stored in the reference face image 
memory 5039 is used as a reference image for detection 
of the face parts. 

[0159] Then, the position information on face parts 
25 detected in the face part detector 5036 is supplied to the 
CPU 51 6A as a signal S5036. 

[0160] Then, in the same way as the initial image, the 
size of the face in the image and the face part detection 
search region of the next image are calculated In the 
30 CPU 51 6A. 

[0161] Figure 1 3 and Fig. 15 show the search regions 
of face parts. 

[01 62] Next, in the CPU 51 6A, the rate of enlargement 
or reouction is calculated from the calculated size of the 

35 face. 

[0163] This rate is that for enlarging or reducing the 
face so that the face Image just fully fills the screen of 
the monitor on which the user's face is displayed. 
[0164] For example, as shown In Fig. 14, since the 
40 size of the face is 3a in the horizontal direction and the 
size of the screen is d in the horizontal direction, the 
reduction rate d/3 is obtained. 

[0165] In addition, in Fig. 16, the enlargement rate d/ 
3a' is obtained. 
45 [0166] The rate calculated by the CPU 51 6A Is sup- 
plied to the image enlarging and reducing unit 5037 as 
a signal S516c. 

[0167] In the image enlarging and reducing unit 5037, 
the input image Is enlarged or reduced based on the sig- 
50 nalS516c. 

[0168] The enlarged or reduced Image is supplied to 
the image extractor 5038. In the Image extractor 5038, 
the face image in the region displayed on the monitor 
screen from the enlarged or reduced image is cut out 
55 based on the face position infomnation (coordinates) 
that has already been sent from the CPU 51 6A as the 
signal S516b. 

[0169] Figure 1 4 and Fig. 1 6 show the cases in which 
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the face Size in the input image is larger than the monitor 
screen and the face size is smaller than the monitor 
screen, respectively. 

[0170] Although the sizes of the faces in the input im- 
ages are different, the sizes of the face images dis- 
played on the monitor screen are the sanr>e. 
[0171] The cut out face image is supplied to the video 
compressor and coder 504 and the later image combin- 
ing circuit 52. 

[0172] Further, the cut out face image is sent to the 
reference face image memory 5039. At this time, a ref- 
erence image is constructed from this Image and the 
rate of enlargement or reduction. 
[0173] According to the second embodiment, similar 
effects as the first embodiment can be obtained. 

Third Embodiment 

(01 74] Figure 1 7 is a circuit diagram of a third embod- 
iment of the signal processing system of a terminal func- 
tioning as a data transmission apparatus according to 
the present invention. 

[01 75] Figure 1 8 is a circuit diagram of an example of 
the specific configuration of the principal portion of the 
pre-processing circuit according to the third embodi- 
ment. 

[0176] ..The point of difference between the signal 
processing system of a terminal according to the third 
embodiment and the aforesaid signal processing sys- 
tem according to the second embodiment lies In the 
processing of the pre-processing circuit 503B and the 
CPU 516B. 

[0177] Specifically, the face image is enlarged or re- 
duced and the face image Is cut out so that the face im- 
age is constantly maintained to a size specified by the 
user. 

[01 78] For this, the pre-processing circuit 503B is con- 
figured to have add a monitor/face Image ratio memory 
5040 to the circuit in Fig. 10. 

[01 79] The rest of the configuration Is the same as that 
shown in Fig. 9 and Fig. 10. Therefore, in Fig. 17 and 
Fig. 18, components the same as those in Fig. 9 and 
Fig. 10 are shown by the same reference numerals. 
[0180] Namely, the signal processing system 500B of 
a terminal, as shown In Fig. 1 7, has a microphone 501 , 
a camera circuit 502, a pre-processing circuit 503B, a 
video compressor and coder 504, a voice'compressor 
and coder 505, a multiplexer (MUX) 506, a transmission 
circuit 507, a reception circuit 508, a demultiplexer (De- 
MUX) 609, a video decoding circuit 51 0, a voice decod- 
ing circuit 511, an image combining apparatus 512, a 
display circuit 51 3, a speaker 514, a GUI interface (l/F) 
circuit 515, and a CPU 51 6B. 

[0181] The pre-processing circuit 503B, as shown in 
Fig. 18, has a face part position detector 5036, an Image 
enlarging and reducing unit 5037, an image extractor 
5038, a reference face Image memory 5039, and a mon- 
itor/face image ratio memory 5040. 



[0182] Because the functions of the pre-processing 
circuit 503B and the CPU 51 6B are different from those 
in the second embodiment, below, only the functions of 
the pre-processing circuit 503B and the CPU 51 SB will 
5 be described in detail. Descriptions of other components 
will be omitted. 

[0183] The monitor/face image ratio memory 5040 
stores values for calculating the size of the face to be 
displayed on the monitor screen and the position of the 
10 face in the monitor screen based on the position Infor- 
mation on face parts extracted by the CPU 51 6B. The 
stored data is used as a reference value in the second 
input image and on. 

[0184] Next, the operation of the pre-processing cir- 
15 cult 503B, Including the processing of the CPU 51 6B, 
will be described with reference to Fig. 19 to Fig. 23. 
[01 85] Rgure 1 9 and Fig. 20 are views for explaining 
processing for reducing a face image In a specified im- 
age transmission region and displaying It on a display 
monitor, Figure 21 is a view for explaining the ratio of 
the pixels "e" efface color over the pixels "f" in the image 
on the monitor, and Fig. 22 and Fig. 23 are views for 
explaining processing for reducing a face image in a 
face part detection search region and displaying it on a 
25 display monitor. 

[01 86] First, the user specifies the region of his or her 
own face desired to be displayed on the monitor (desired 
to be transmitted) in the first Input image (the Initial im- 
age) on for example the GUI of the display monitor. This 
30 information on the region is supplied to the CPU 51 6B. 
[01 87] The region is easy for the user to specify is re- 
sembling the monitor. 

[0188] In the CPU 51 SB, the supplied Initial region is 
supplied to the face part position detector 5036 as a face 

35 part detection search region signal S516a. 

[01 89] In the face part position detector 5036, the po- 
sition of parts of the face such as the two eyes, nose, 
mouth, eyebrows, ears, and outline of the face Is detect- 
ed in the supplied face part search region. The position 

40 infomnation on face parts detected in the face part de- 
tector 5036 is supplied to the CPU 51 SB as a signal 
S5036. 

[0190] In the CPU 51 SB, the size of the face displayed 
on the monitor screen and values for calculating the po- 

45 sition of the face in the monitor are extracted based on 
the position information of face parts. 
[01 91] For example, in the first Input Image (the Initial 
Image), as shown In Fig. 20, the horizontal length "c" of 
the region specified by the user, the distance "a" be- 

50 tween the detected two eyes in the initial image, and the 
horizontal length "d" of the display monitor 53 can be 
used. 

[0192] Further, as shown in Fig. 21, it is possible to 
detect the face color and calculate the ratio of the pixels 
55 "e" of face color over the pixels T' in the image on the 
monitor. 

[0193] These values are sent to the ratio memory 
5040 and used as reference values from the second in- 
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put image and on. 

[0194] Note these values may also be stored in the 
CPU516B. 

[0195] Further, in the CPU 51 6B, the face part detec- 
tion search region of the next image Is calculated based 
on the position infomnation of the face parts. 
[0196] For example, in the same way as the second 
embodiment, as shown In Fig. 1 2, assume the region of 
±a X N horizontally and +b x M and -b x L vertically 
around the rectangle a x b including the two eyes and 
the mouth is the face part detection search region of the 
next image. 

[0197] For the second input image and on. the face 
part detection search region obtained In the previous im- 
age is supplied to the face part position detector 5036 
by the CPU 51 6B. 

[0198] In the face part position detector 5036, posi- 
tions of parts of the face are detected in the supplied 
search region In the same way as the above. 
[0199] At this time, Information of the face portion of 
the previous Image stored in the reference face image 
memory 5039 is used as a reference Image for detection 
of parts of the face. 

[0200] The position infomnation on face parts detect- 
ed in the face part detector 5036 Is supplied to the CPU 
51 SB as a signal S5036. 

[0201 ] Next, in the CPU 51 SB. the rate of enlargement 
or reduction is calculated from the calculated size of the 
face. 

[0202] In the first image (initial image), for example, 
the reduction rate d/c Is obtained from the horizontal 
length "c: of the region specified by the user and the 
horizontal length "d: of the screen stored in the ratio 
memory 5040. 

[0203] For the second image and on, the ratio is cal- 
culated from the face position infomnation sent from the 
face part position detector 5036 and the reference val- 
ues stored in the ratio memory 5040. For example, as 
shown in Fig. 23, the reduction rate aVa Is calculated 
from the distance a' between the two eyes sent from the 
face part position detector 5036 and the distance "a" be- 
tween the two eyes in the first image stored in the ratio 
memory 5040 as a reference value. 
[0204] Further, the reduction rate may also be found 
using the skin color pixels "g" from the face part position 
detector 5036 and the skin color pixels "e" In the initial 
image stored in the ratio memory 6040. 
[0205] The rate calculated by the CPU 51 6B is sup- 
plied to the image enlarging and reducing unit 5037 as 
a signal S516c. 

[0206] In the image enlarging and reducing unit 5037. 
the input image is enlarged or reduced based on the sig- 
nal S516C. 

[0207] The enlarged or reduced image is supplied to 
the image extractor 5038. In the image extractor 5038, 
the face image in the region displayed on the monitor 
screen from the enlarged or reduced image is cut out 
based on the face position Infomnation (coordinates) al- 



ready sent from the CPU 51 SB as the signal S516b. 
[0208] Figure 23 shows the state where the size of the 
face In the input image which is larger than the monitor 
screen is reduced so as to be displayed at the same as 
5 the face size specified by the user in the initial Image 
and where the size of the face image displayed in the 
monitor screen becomes always the same. 
[0209] When the face In the input image is smaller 
than the initial image, it will be enlarged. 
10 [0210] The extracted face image is supplied to the lat- 
er video compressor and coder 504 and image combin- 
ing circuit 52. 

[021 1 ] Further, the extracted face image is sent to the 
reference face image memory 5039. At this time, a ref- 
15 erence image is constructed from this image and the 
rate of enlargement or reduction. 
[0212] According to the third embodiment, similar ef- 
fects as the first embodiment can be obtained. 



20 Fourth Embodiment 

[0213] Next, a specific example of improving the 
tracking accuracy of an image by fetching and tracking 
the image at a frame rate the same as or higher than 
25 the image to be transmitted and refreshing a reference 
image serving as a reference for tracking by a rate the 
same as or higher than the transmission frame rate will 
be explained as a fourth embodiment of the present in- 
vention. 

30 [0214] Figure 24 is a conceptual view for explaining 
the processing for improving the tracking accuracy of an 
image by fetching and tracking the image at a frame rate 
the same as or higher than the image to be transmitted 
and refreshing a reference image serving as a reference 
35 for tracking by a rate the same as or higher than the 
transmission frame rate 

[0215] Note that Fig. 24 shows the case in which the 
frame rate of the image to be transmitted Is one-third the 
input frame rate. In practice, the frame rate ratio is con- 
40 trolled according to the bandwidth of the transmission 
channel and the congestion state and is not set con- 
stant. 

[0216] In the example of Fig. 24, each input frame is 
refreshed using the face Image of the previous image 
45 as a reference Image and matching is perfomned In the 
search region of the next Image. 
[0217] For example, a face image detected in 100-1 
of Fig. 24 is used as the reference image 1 00-8 of the 
next image, and matching is performed for 100-8 in the 
50 image 100-2 to detect the face image. Then, the face 
image detected here Is used as the next reference im- 
age. 

[0218] Further, from the position of this face image, 
the search region of the next input image is decided and 
55 refreshed. 

[0219] In accordance with the frame rate of the image 
to be transmitted, the image to be transmitted is selected 
from the face image detected in the input frame. 
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[0220] In Fig. 24, face images cut out In images 1 00-1 , 
100-4, and 100-7 are transmitted. 
[0221] When setting the frame rate of image transmis- 
sion the same as that of reference image refreshing, the 
time Interval between the reference image and the Im- 
age to be matched becomes longer, therefore there is 
a possibility that the matching accuracy will become 
poor for example if the face Image becomes gradually 
smaller, the expression changes drastically, or the face 
is turned at a slant. 

[0222] For example, when matching 1 00-8 and 1 00-4, 
the sizes of the face images are different, so the match- 
ing accuracy is poor. In addition, when setting the rate 
of the refreshing of the search region the same as the 
frame rate of the image to be transmitted, if the face is 
moved horizontally, sometimes the face image will leave 
the search region, so there is a possibility the matching 
accuracy will become poor. 

[0223] For example, when using the search region of 
100-1 for matching 100-4, the face image will not be cov- 
ered by the search region at ail, so the matching accu- 
racy will be poor 

[0224] As in the fourth embodiment, if the image is 
fetched and tracked at a frame rate the same as or high- 
er than the Image to be transmitted and a reference im- 
age serving as a reference for tracking Is refreshed at a 
rate the same as or higher than the transmission frame 
rate, the tracking accuracy can be improved. 
[0225] In embodiments of the present invention, the 
total amount of Infomriation can be greatly reduced while 
transmitting the expression of the other party which is 
needed most for communication. 
[0226] In addition, in embodiments the present inven- 
tion,. because the face image is adjusted to fully fill the 
frame, transmission of the high resolution background 
image can be naturally eliminated, and the amount of 
information after compression can be largely reduced. 
Consequently, It is possible to transmit an Image of a 
high quality. 

[0227] Summarizing the effects of embodiments of 
the Invention, the frame correlation of Images is able to 
be Increased greatly. By using a compression technique 
of Interframe compression like MPEG for the image 
compression method, the amount of Information can be 
decreased drastically, and therefore it becomes possi- 
ble to transmit an Image of a high quality. 
[0228] In addition. In an embodiment of the present 
Invention, the image state that a user desires to transmit 
(image locking state) can be kept automatically. This 
helps detect the characteristic portion of the Image and 
enables tracking of the target image such as the Image 
of a person. As a result, the time correlation of images 
Is increased, and the amount of information after com- 
pression can be decreased. Therefore it becomes pos- 
sible to transmit an image of a high quality. 
[0229] Further, in an embodiment of the present in- 
vention, a state can be realized in which the users main- 
tain better eye contact. 



[0230] Further, in an embodiment of the present in- 
vention, the state of the users talking white viewing an 
Image for discussion can be transmitted. Further, there 
is an advantage that virtual reality communication where 

5 the users maintain eye contact by looking at each oth- 
ers? faces on the monitor can be realized. 
[0231] While the invention has been described with 
reference to specific embodiment chosen for purpose of 
illustration, it should be apparent that numerous modifi- 

10 cation could be made thereto by those skilled in the art 
without departing from the basic concept and scope of 
the invention. 



15 Claims 



A data transmission method for transmitting at least 
image data accommodated In an image frame of a 
preset size, comprising the steps of: 

capturing an image including a target image to 
be transmitted; 

making adjustments so that the target Image to 
be transmitted from the captured image sub- 
stantially fully fills the Image frame; and 
compressing and transmitting the adjusted im- 
age data In the Image frame. 



20 



25 



2. A data transmission method as set forth in claim 1 , 
30 wherein when making adjustments so that a target 

image fully fills an image frame, comphsing using 
an Image of a characteristic portion of the image to 
be transmitted for matching and cutting out the Im- 
age around the part with the best match for tracking 
35 of the image. 

3. A data transmission method as set forth In claim 1 , 
wherein when making adjustments so that a target 
image fully fills an image frame, comphsing using 

40 an Image of a characteristic portion of the Image to 
be transmitted to calculate a distance In the char- 
acteristic portion and using this for enlargement, re- 
duction, and tracking of the Image. 



45 



50 



55 



5. 



A data transmission method as set forth in claim 2, 
wherein when detecting a characteristic portion of 
an image, comprising locking an image in a desired 
image state while monitoring a state of input of the 
image to be transmitted and using a center portion 
of the locked image as the characteristic portion of 
the image. 

A data transmission method as set forth in claim 3, 
wherein when detecting a characteristic portion of 
an image, comprising locking an image in a desired 
Image state while monitoring a state of Input of the 
image to be transmitted and using a center portion 
of the locked image as the characteristic portion of 
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the image. 

6. A data transmission metliod as set forth In claim 2, 
wherein when detecting a characteristic portion of 
an image, comprising displaying a state of input of 
the Image to be transmitted, specifying a certain lo- 
cation on the display screen, and using the portion 
around the specified point as the characteristic por- 
tion of the image. 

7. A data transmission method as set forth in claim 3, 
wherein when detecting a characteristic portion of 
an Image, comprising displaying a state of input of 
the image to be transmitted, specifying a certain lo- 
cation on the display screen, and using the portion 
around the specified point as the characteristic por- 
tion of the Image. 

8. A data transmission method as set forth in claim 4, 
wherein when locking a target image by using a 
characteristic portion of an image, comprising fetch- 
ing and tracking the image at a frame rate the same 
as or higher than the Image to be transmitted and 
refreshing a reference image serving as a reference 
for tracking at a rate the same as or higher than the 
transmission frame rate. 



9. A data transmission method as set forth in claim 5, 
wherein when locking a target Image by using a 
characteristic portion of an image, comprising fetch- 
ing and tracking the Image at a frame rate the same 
as or higher than the image to be transmitted and 
refreshing a reference image sen/ing as a reference 
for tracking at a rate the same as or higher than the 
transmission frame rate. 

10. A data transmission method as set forth in claim 1 . 
comprising displaying a state of input of the image 
to be transmitted and specifying a range to be trans- 
mitted on the display to determine the range of 
transmission of the Image. 

1 1 . A data transmission method for transmitting at least 
image data, comprising the steps of: 

capturing an image Including a target image to 
be transmitted; 

making adjustments so that the target image to 
be transmitted from the captured image has a 
preset size; and 

compressing and transmitting the adjusted im- 
age data in the image frame. 

1 2. A data transmission method as set forth in claim 1 1 , 
wherein when making adjustments so that a target 
image has a preset size, comprising using an image 
of a characteristic portion of the image to be trans- 
mitted for matching and cutting out the image 



around the part of the best match for tracking of the 
image. 

1 3. A data transmission method as set forth in claim 1 1 . 
5 wherein when making adjustments so that a target 

image has a preset size, comprising using an image 
of a characteristic portion of the image to be trans- 
mitted to calculate a distance in the characteristic 
portion and using this for enlargement, reduction, 
10 and tracking of the image. 

14. A data transmission method as set forth in claim 1 2, 
wherein when detecting a characteristic portion of 
an image, comprising locking an image in a desired 

15 image state while monitoring a state of input of the 
image to be transmitted and using a center portion 
of the locked Image as the characteristic portion of 
the Image. 

20 15. A data transmission method as set forth in claim 13. 
wherein when detecting a characteristic portion of 
an image, comprising locking an image in a desired 
image state while monitoring a state of input of the 
image to be transmitted and using a center portion 

25 of the locked image as the characteristic portion of 
the image. 

16. A data transmission method as set forth in claim 1 2, 
wherein when detecting a characteristic portion of 

30 an image, comprising displaying a state of input of 
the image to be transmitted, specifying a certain lo- 
cation on the display screen, and using the portion 
around the specified point as the characteristic por- 
tion of the image. 

35 

17. A data transmission method as set forth In claim 13, 
wherein when detecting a characteristic portion of 
an image, comprising displaying a state of input of 
the image to be transmitted, specifying a certain lo- 

40 cation on the display screen, and using the portion 
around the specified point as the characteristic por- 
tion of the Image. 



1 8. A data transmission method as set forth in claim 1 4, 
45 Wherein when locking a target Image by using a 

characteristic portion of an Image, comprising fetch- 
ing and tracking the image at a frame rate the same 
as or higher than the image to be transmitted and 
refreshing a reference image serving as a reference 
50 for tracking at a rate the same as or higher than the 
transmission frame rate. 

19. A data transmission method as set forth in claim 15, 
Wherein when locking a target image by using a 

55 characteristic portion of an image, comprising fetch- 
ing and tracking the image at a frame rate the same 
as or higher than the Image to be transmitted and 
refreshing a reference image serving as a reference 
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for tracking at a rate the same as or higher than the 
transmission frame rate. 

20. A data transmission method as set forth in claim 1 1 , 
comprising displaying a state of input of a transmit- 
ted image and specifying a range to be transmitted 
on the display to detemnine the range of transmis- 
sion of the image. 

21. A data transmission method for displaying at least 
image data of a user on a display screen among a 
plurality of tenninals and transmitting at least image 
data while displaying received Image data on the 
display screen, comprising capturlng'an image in- 
eluding a target image of a user to be transmitted 
from substantially the center of the display screen. 

22. A data transmission method for displaying at least 
image data of user on a display screen among a 
plurality of terminals and transmitting at least image 
data while displaying received image data of anoth- 
er party on the display screen and displaying con- 
tent for common discussion on the display screen, 
comprising the steps pt 

capturing an image including a target image of 
a user to be transmitted from substantially a 
■center of the display screen and 
. idisplaying the image data at one of an upper 
portion or a lower portion of the display screen 
from the substantially center portion of the dis- 
. ...play screen serving as the capturing portion 
. and displaying the content at the other portion. 

23. A data transmission method as set forth in claim 22, 
comprising displaying image data above the con- 
tent on the display screen. 

24. A data transmission method as set forth In claim 22, 
comprising displaying image data below the content 
on the display screen. 

25. A data transmission apparatus for transmitting at 
least image data accommodated In an Image frame 
of a preset size, comprising 

an imaging means for capturing an image in- 
cluding a target image to be transmitted; 
a first circuitfor making adjustments so that the 
target image to be transmitted from the cap- 
tured image by the imaging means substantial- 
ly fully fills the image frame; and 
a second circuit for compressing and transmit- 
ting the adjusted image data in the image 
frame. 

26. A data transmission apparatus as set forth In claim 
25, wherein when making adjustments so that the 



target image to be transmitted fully fills the Image 
frame, said first circuit uses an image of a charac- 
teristic portion of the image to be transmitted for 
matching and cuts out the image around the part 
5 with the best match for tracking of the Image. 

27. A data transmission apparatus as set forth in claim 

25, wherein when making adjustments so that the 
target image to be transmitted fully fills the Image 
frame, said first circuit uses an image of a charac- 
teristic portion of the Image to be transmitted to cal- 
culate a distance in the characteristic portion and 
uses this for enlargement, reduction, and tracking 
of the image. 

28. A data transmission apparatus as set forth in claim 

26, comprising 

a displaying means for displaying image data 
and 

a locking means able to lock an image dis- 
played by the displaying means in a desired im- 
age state, wherein 

when detecting a characteristic portion of an 
image, said first circuit uses the center portion 
of the image locked by the locking image as the 
characteristic portion of the image. 

29. A data transmission apparatus as set forth In claim 

27, comprising 

a displaying means for displaying image data 
and 

a locking means able to lock an image dis- 
played by the displaying means in a desired 
state, wherein 

when detecting a characteristic portion of an 
image, said first circuit uses the center portion 
of the Image locked by the locking image as the 
characteristic portion of the Image. 

30. A data transmission apparatus as set forth in claim 

26, comprising 

a displaying means for displaying Image data 
and including a pointer able to specify a certain 
location and 

a specifying means able to specify a certain lo- 
cation of the Image displayed on the displaying 
means by the pointer, wherein 
when detecting a characteristic portion of an 
image, said first circuit uses a portion around 
the point specified by the specifying means as 
the characteristic portion of the image. 

31 . A data transmission apparatus as set forth in claim 

27. comprising 
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a displaying means for displaying image data 
and including a pointer able to specify a certain 
location and 

a specifying means able to specify a certain lo- 
cation of the image displayed on the displaying 
means by the pointer, wherein 
when detecting a characteristic portion of an 
image, said first circuit uses a portion around 
the point specified by the specifying nneans as 
the characteristic portion of the image. 

32. A data transmission apparatus as set forth in claim 

28, wherein when locking a target Image by using 
a characteristic portion of the image, said first circuit 
fetches the image and perfomis a tracl<ing opera- 
tion at a frame rate the same as or higher than the 
Image to be transmitted and refreshes a reference 
Image serving as a reference for tracking at a rate 
the same as or higher than the transmission frame 
rate. 

33. A data transmission apparatus as set forth in claim 

29. wherein when locking a target image by using 
a characteristic portion of the image, said first circuit 
fetches the image and perfonns a tracking opera- 
tion at a frame rate the same as or higher than the 
image to be transmitted and refreshes a reference 
image serving as a reference for tracking at a rate 
the same as or higher than the transmission frame 
rate. 



34. A data transmission apparatus as set forth in claim 
25, comprising 

a displaying means for displaying image data 
and including a pointer able to specify a certain 
location and 

a specifying means able to specify a certain lo- 
cation of the image displayed by the displaying 
means by the pointer, wherein 
said first circuit decides on the range specified 
by the specifying means as the range for trans- 
mission of the image. 

35. A data transmission apparatus for transmitting at 
least image data, comprising 

an imaging means for capturing an Image in- 
cluding a target image to be transmitted; 
a first circuit for making adjustments so that the 
target image to be transmitted from the cap- 
tured image has a preset size; and 
a second circuit for compressing and transmit- 
ting the adjusted image data in the image 
frame. 

36. A data transmission apparatus as set forth in claim 
35. wherein when making adjustments so that the 



target image has a preset size, said first circuit uses 
an image of a characteristic portion of the image to 
be transmitted for matching and cuts out the image 
around the part with the best match for tracking of 
5 the image. 

37. A data transmission apparatus as set forth in claim 

35, wherein when making adjustments so that the 
target image has a preset size, said first circuit uses 

10 an image of a characteristic portion of the image to 
be transmitted to calculate a distance In the char- 
acteristic portion and uses this for enlargement, re- 
duction, and tracking of the image. 

15 38. A data transmission apparatus as set forth in claim 

36, comprising 

a displaying means for displaying image data 
and 

20 a locking means able to lock an image dis- 

played by the displaying means In a desired 
state, wherein 

when detecting a characteristic portion of an 
image, said first circuit uses a center portion of 
25 the image locked by the locking image as the 

characteristic portion of the image. 

39. A data transmission apparatus as set forth in claim 

37, comprising 

a displaying means for displaying image data 
and 

a locking means able to lock an image dis- 
played by the displaying means in a desired 
state, wherein 

when detecting a characteristic portion of an 
image, said first circuit uses a center portion of 
the image locked by the locking image as the 
characteristic portion of the image. 

40. A data transmission apparatus as set forth in claim 
36, comprising 

a displaying means for displaying image data 
and Including a pointer able to specify a certain 
location and 

a specifying means able to specify a certain lo- 
cation of the image displayed on the displaying 
means by the pointer, wherein 
when detecting a characteristic portion of an 
image, said first circuit uses a portion around 
the point specified by the specifying means as 
the characteristic portion of the image. 
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and including a pointer able to specify a certain 
location and 

a specifying means able to specify a certain lo- 
cation of the image displayed on tlie displaying 
means by the pointer, wherein 
when detecting a characteristic portion of an 
image, said first circuit uses a portion around 
the point specified by the specifying means as 
the characteristic portion of the image. 

42. A data transmission apparatus as set forth in claim 

38, wherein when locking a target image by using 
a characteristic portion of the image, said first circuit 
fetches the image and perfomis a tracking opera- 
tion at a frame rate the same as or higher than the 
image to be transmitted and refreshes a reference 
Image serving as a reference for tracking at a rate 
the same as or higher than the transmission frame 
rate. 

43. A data transmission apparatus as set forth in claim 

39, wherein when locking a target image by using 
a characteristic portion of the image, said first circuit 
fetches the image and performs a tracking opera- 
tion at a frame rate the same as or higher than the 
image to be transmitted and refreshes a reference 
image sen/ing as a reference for tracking at a rate 
the same as or higher than the transmission frame 
rate. 

44. A data transmission apparatus as set forth In claim 
35, comprising 

a displaying means for displaying image data 
and including a pointer able to specify a certain 
location and 

a specifying means able to specify a certain lo- 
cation of the image displayed by the displaying 
means by the pointer, wherein 
said first circuit decides on the range specified 
by the specifying means as the range for trans- 
mission of the Image. 

45. A data transmission system for displaying at least 
image data of a user on a display screen among a 
plurality of terminals and transmitting at least Image 
data while displaying received Image data of anoth- 
er party on the display screen, wherein a terminal 
comprises 

an Imaging means for capturing an image in- 
cluding a target image to be transmitted; 
a first circuit for making adjustments so that the 
target image to be transmitted from the cap- 
tured image by the imaging means substantial- 
ly fully fills an Image frame; and 
a second circuit for compressing and transmit- 
ting the adjusted image data In the Image 



frame. 

46. A data transmission system for displaying at least 
Image data of a user on a display screen among a 

5 plurality of terminals and transmitting at least image 
data while displaying received image data of anoth- 
er party on the display screen, wherein a terminal 
comprises 

^0 an imaging means for capturing an image in- 

cluding a target image to be transmitted; 
a first circuit for making adjustments so that the 
target image to be transmitted from the cap- 
tured image becomes a preset size; and 
^5 a second circuit for compressing and transmit- 

ting the adjusted image data in an image frame. 

47. A data transmission system for displaying at least 
Image data of a user on a display screen among a 
plurality of terminals and for transmitting at least Im- 
age data while displaying received image data on 
the display screen, wherein 

an Imaging means for capturing an image In- 
cluding a target image to be transmitted is provided 
substantially at the center of the display screen. 

48. A data transmission system for displaying at least 
image data of a user on a display screen among a 
plurality of terminals and for transmitting at least im- 
age data while displaying received ^age data of 
another party on the display screen and displaying 
content for common discussion on the display 
screen comprising 

an imaging means provided substantially at the 
center of the display screen and capturing an 
Image Including a target image to be transmit- 
ted and 

a displaying means for displaying the image da- 
ta at one of an upper portion or a lower portion 
of the display screen from the substantially 
center portion of the display screen serving a-i 
the capturing portion and displaying the contient 
at the other portion. 
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FIG.1 1 




EXAMPLE OF FACE SIZE CALCULATION 
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FIG. 12 




EXAMPLE OF FACE PART DETECTION SEARCHING REGION 
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FIG. 19 



IMAGE TRANSMISSION 
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FIG.21 

PIXELS IN SPECIFIED INITIAL REGION 
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